As new data collection technologies have evolved, allowing the analysis of more complex signals, techniques for analyzing such data have become more complex. For example, the introduction of new technologies in spectroscopy has allowed the collection of very large amounts of data. In some spectroscopic measurements, a high degree of sensitivity is needed to sense constituents that have relatively low concentrations or that lack highly selective spectral attributes. Devices used in such measurements, however, can capture extraneous signals, which can interfere with the accurate extraction of the desired information. For example, Johnson noise and shot noise, fundamental noise sources present in virtually all optically-based instruments, are often-cited sources of extraneous information. Data capture and analysis techniques can reduce the deleterious effects of such noise sources. For some random noise sources, simply extending measurement times or further optimizing the electronics or instrumental set-up can eliminate noise effects.
There are additional non-fundamental noise sources that can be difficult to model in advance that originate in the measurement device, or in the various interfaces to the device. For example, in infrared or near-infrared Raman spectroscopy, a sample is illuminated using an optical source, and light reflecting from or transmitted through the sample is gathered and analyzed to determine characteristics of the sample. The sensitivity of optical detection elements can change, and the output of the excitation source can change. These are two examples of non-fundamental noise sources. Extending measurement times and co-averaging multiple measurements do not always mitigate non-fundamental noise sources.
One traditional approach to eliminating such artifacts is to make measurements of a reference sample, or background, which ostensibly provides a constant measurement response over time. In some instantiations the reference sample is not a sample at all, but rather a measurement in the absence of any sample, sometimes referred to as a ‘blank’. If the measured response of this reference is observed to change, it can be inferred that the character of the instrument response has itself changed. In a common implementation in spectroscopy, a measured spectrum of the background serving as the reference is subtracted from the measured absorbance spectrum of a sample, hypothetically eliminating instrumental or environmental artifacts that have commonly corrupted both the reference and sample measurements. Mathematically, this can be expressed as in equations 1 and 2:xsamp=xsampo+δs  (1)xref=xrefo+δr  (2)where xsamp is the observed sample spectrum, xref is the observed reference spectrum, and the superscript ‘o’ denotes the true (but unobservable) signal character of the sample or reference as indicated. The δ terms appearing in equations 1 and 2 are the instrumental or environmental noise disturbances distorting the measurements. The correction procedure involving background subtraction (or ratioing when operating in intensity rather than absorbance) entails{tilde over (x)}samp=xsamp−xref  (3)
Provided the measurement distortions are equivalent, the following relations hold:{tilde over (x)}samp=(xsampo+δs)−(xrefo+δr)  (4){tilde over (x)}samp=xsampo−xrefo  (5)
Thus, the resultant spectrum {tilde over (x)}samp includes only signal characteristics of the sample and reference, and not distortions in the measurements associated with the measurement device or the sampling environment. There can be aspects of the distortions which do not subtract (e.g., photon shot noise, detector noise), but background subtraction or ratioing does not set out to eliminate these distortions.
In the application of the technique described above it is assumed that δs=δr, and that the reference, xrefo, will not vary; hence, any observed change in {tilde over (x)}samp must be exclusively attributable to a change in the sample. With a collection of background-corrected sample measurements, any number of known multivariate techniques can be used to disassemble the useful signal and, if desired, further determine the sample characteristics of interest. FIG. 1 illustrates this approach result graphically. For instance, a multivariate regression model can be generated from background-corrected spectra to estimate sample properties.
FIG. 2 demonstrates the application of the approach described above when the distortions in the two measurements are non-equivalent, even under scalar multiplication by k:δs≠kδr  (6)If the distortion were simply multiples then conventional background subtraction could still work with trivial modifications. In this case the subtraction of the reference measurement from the sample does not result in the cancellation of the distortion in the sample spectrum.xsamp=xsampo+δs  (7)xref=xrefo+δr  (8)
The resulting corrected spectrum {tilde over (x)}samp is still corrupted by the measurement disturbances:{tilde over (x)}samp=(xsampo+δs)−(xrefo+δr)  (9){tilde over (x)}samp=(xsampo+xrefo)−(δs+δr)  (10)
The possible causes of non-equivalence between δs and δr are myriad. Detector non-linearity, and differences in absorption and scattering properties are typical culprits, but there are many others. There is a need for improved techniques for determining accurate measurements in the presence of such distortions.